Location signaling with respect to an autonomous vehicle and a rider

ABSTRACT

Among other things, a signal is emitted from a signaling device and is captured by stimulus detectors location on a vehicle, including an autonomous vehicle. Properties of the signal are analyzed, potentially in concert with other information, to determine the precise location of the signaling device and therefore the precise goal location for picking up a rider of the autonomous vehicle. In response to the calculation of the precise goal location, the autonomous vehicle attempts to travel to a location as near as possible to the precise goal location to facilitate entry of the rider into the vehicle.

BACKGROUND

This description relates to location signaling with respect to anautonomous vehicle (sometimes abbreviated as an AV) and a rider.

As shown in FIG. 1, in a common method for a rider of a taxicab or aride-sharing vehicle to hail and then enter the vehicle, the rider posts10 a ride request by interacting with a mobile app. The ride requestincludes information about a general goal location for the pick-updesired by the rider, which is often specified imprecisely orinaccurately. The ride request is processed by the mobile app and thenforwarded 12 through the cellular telephone network or Internet to acentral server where it is further processed (including potentiallygeocoding the pick-up location into coordinates and assigning a vehicle)and then forwarded 14 through the cellular telephone network or Internetto the driver of the assigned vehicle. The driver of the assignedvehicle travels 16 to the rider's desired general goal location, thenoften communicates by phone or text or both with the rider 18 to learnor agree upon a precise goal location for the pick-up activity, which isa location that is often defined precisely and accurately. The driver ofthe vehicle then travels 20 to the rider's precise goal location and therider is authenticated and enters the vehicle 22 which then proceeds toa destination 24.

SUMMARY

In general, in an aspect, when an autonomous vehicle is in the vicinityof a general goal location for a goal location activity, the autonomousvehicle or a user or both of them engages in location signalingactivities. Based on information determinable from the locationsignaling activities, the autonomous vehicle or the user or both move toa stopping place that is determined to be feasible and acceptable and atwhich the goal location activity can occur. In some implementations, aprecise goal location is determined. The stopping place is in thevicinity of the precise goal location.

In general, in an aspect, when an autonomous vehicle is in the vicinityof a general goal location for a goal location activity, the autonomousvehicle or a user or both of them engage in location signalingactivities. Based on information determinable from the locationsignaling activities, a process determines a precise goal location.

Implementations may include one or a combination of two or more of thefollowing features. The process determines a stopping place in thevicinity of the precise goal location. An AV system associated with theautonomous vehicle determines the precise goal location. The locationsignaling activities include the user or a signaling device associatedwith the user sending explicit precise goal location information. Theprecise goal location is inferred from the signaling activities withoutthe signaling activities conveying explicit location information Thesignaling activities include the user or a signaling device associatedwith the user sending explicit precise goal location information.Signaling activities include the autonomous vehicle sendingacknowledgment information with respect to the precise goal location.The signaling activities include the autonomous vehicle sendinginformation about its location. The location signaling activitiesinclude line of sight signaling. The location signaling activitiesinclude sending messages through a central server. The precise goallocation is in the vicinity of the general goal location. The signalingactivities convey information about a precise goal location in thevicinity of the general goal location. The user hails the movingautonomous vehicle when it is in the vicinity of the user. The locationsignaling activities engaged in by the user include controlling anapplication running on a mobile device. The location signalingactivities engaged in by the user include manually effectednon-electronic signaling activities. The location signaling activitiesinvolve a location indication signal. The autonomous vehicle moves to oroccupies a target stopping place close to the precise goal location. Theprecise goal location is the actual precise location of the user. Anidentification of the general goal location is received from the user.The identification of the general goal location is received from aprocess. The identification of the general goal location is receivedfrom an external source at the autonomous vehicle. A process determinesa precise goal location based on the location signaling activities. Theprecise goal location is determined based on a location indicationsignal sent by the user or a device of the user. The precise goallocation is determined based on information received other than by thelocation signaling activities. The location signaling activities includea location response signal sent by or on behalf of the autonomousvehicle. The location signaling activities include signals indicatingprogress of the autonomous vehicle toward the precise goal location. Thelocation signaling activities include signaling modes that include oneor a combination of two or more of wireless signaling or non-wirelesssignaling. The wireless signaling modes include electronic signaling ornon-electronic signaling. The non-electronic signaling includes one or acombination of two or more of displaying an image or sequence of images,emitting a sequence of light signals, emitting a sequence of soundsignals, emitting a wireless communication signal, or engaging ingestures or facial expressions. The electronic signaling includessignaling from one or a combination of two or more of a smartphone, atablet, a smart watch, smart glasses, or other smart wearable device.The non-electronic signaling includes one or a combination of two ormore of a hand, or facial, or other gesture, or whistling, yelling, ormaking another sound. non-electronic signaling involves identifiablepassive features of the user. The location signaling activities conformto a commonly accepted protocol. The process bases the determination ofthe precise goal location on a known signaling mode having been used atleast in part for the location signaling activities. The processdetermines a bearing from the autonomous vehicle. The bearing isdetermined repeatedly. The process determines a distance from theautonomous vehicle. The distance is determined repeatedly. The processuses road data. The location signaling activities include a signalingdevice associated with the user sending its precise location. Theprecise location is sent through a communication network. The preciselocation is determined by the signaling device associated with the user.The location signaling activities include the autonomous vehicle sendingby line-of-sight a location indication signal identifying the locationof the autonomous vehicle. The location signaling activities include theautonomous vehicle sending a signal identifying a precise goal locationdetermined by the autonomous vehicle and in the vicinity of the generalgoal location. The precise goal location changes over time. Anothervehicle receives signals that are part of the location signalingactivities and communicates with the autonomous vehicle or a user devicebased on the signals. A detector in an infrastructure element receivessignals that are part of the signaling activities and communicates withthe autonomous vehicle or a user device based on the signals.

In general, in an aspect, a user of an autonomous vehicle that isproceeding toward a general goal location signaling to the autonomousvehicle information from which a precise goal location in the vicinityof the general goal location can be determined. In some implementations,a target stopping place is determined from the information.

In general, in an aspect, an autonomous vehicle that is proceedingtoward a general goal location signals to a user of the autonomousvehicle information from which a precise goal location in the vicinityof a general goal location can be determined. In some implementations, atarget stopping place is determined from the information.

In general, in an aspect, when an autonomous vehicle is in the vicinityof a general goal location for a goal location activity, the autonomousvehicle or a user or both of them engages in location signalingactivities. Information determinable from the location signalingactivities is presented at a remote device to a remotely locatedteleoperator. Information specified by the remotely located teleoperatoris received from the remote device identifying a precise goal locationin the vicinity of the general goal location.

In general, in an aspect, the autonomous vehicle system determines aprecise goal location in the vicinity of a general goal location towhich an autonomous vehicle is proceeding, a stopping place at which theautonomous vehicle and a user will engage in a goal location activitybeing in the vicinity of the precise goal location.

Implementations may include one or a combination of two or more of thefollowing features. The precise goal location is the actual preciselocation of the user. The precise goal location is inferred from thesignaling activities without the signaling activities conveying explicitlocation information. An identification of the general goal location isreceived from the user. The identification of the general goal locationis received from a process. The identification of the general goallocation is received from an external source at the autonomous vehicle.The determination of the precise goal location is based on a locationindication signal sent by the user or a device of the user. Thedetermination of the precise goal location is based on informationreceived other than by the location signaling activities. The processbases the determination of the precise goal location on a knownsignaling mode having been used at least in part for the locationsignaling activities a position determination process. The processdetermines a bearing from the autonomous vehicle. The bearing isdetermined repeatedly. The process determines a distance from theautonomous vehicle. The distance is determined repeatedly. The processuses road data. The precise goal location changes over time.

In general, in an aspect, the autonomous vehicle system determines aprecise location of a user in the vicinity of a general goal location towhich the autonomous vehicle is proceeding, the precise location beingdetermined based on wireless line-of-sight communication between theautonomous vehicle and the user.

Implementations may include one or a combination of two or more of thefollowing features. The precise goal location is the actual preciselocation of the user. An identification of the general goal location isreceived from the user. An identification of the general goal locationis received from a process. The identification of the general goallocation being is received from an external source at the autonomousvehicle. The precise goal location is determined based on a locationindication signal sent by the user or a device of the user. Thedetermination of the precise goal location is based on informationreceived other than by the location signaling activities. The processbases the determination of the precise goal location based on a knownsignaling mode having been used at least in part for the locationsignaling activities a position determination process. The processdetermines a bearing from the autonomous vehicle. The bearing isdetermined repeatedly. The process determines a distance from theautonomous vehicle. The distance is determined repeatedly. The processuses road data. The precise goal location changes over time. Thesignaling activities include the user or a signaling device associatedwith the user sending precise goal location information. Thecommunication includes the autonomous vehicle sending information aboutits location. communication includes line of sight signaling. Thecommunication includes sending messages through a central server.communication conveys information about a precise goal location in thevicinity of the general goal location. The communication includescontrolling an application running on a mobile device. communicationincludes manually effected non-electronic signaling activities.

In general, in an aspect, two or more autonomous vehicles cooperativelydetermine a precise goal location of a user in the vicinity of a generalgoal location to which one of the autonomous vehicles is proceeding.

Implementations may include one or a combination of two or more of thefollowing features. The precise goal location is in the vicinity of thegeneral goal location. The precise goal location is the actual preciselocation of the user. The identification of the general goal location isreceived from the user. The identification of the general goal locationis received from a process. The identification of the general goallocation is received from an external source at the autonomous vehicle.The precise goal location is determined based on a location indicationsignal sent by the user or a device of the user. The precise goallocation is based on information received other than by the locationsignaling activities. The process bases the determination of the precisegoal location based on a known signaling mode having been used at leastin part for the location signaling activities a position determinationprocess. The process determines a bearing from the autonomous vehicle.The bearing is determined repeatedly. The process determines a distancefrom the autonomous vehicle. The distance is determined repeatedly. Theprocess uses road data. The precise goal location changes over time.

In general, in an aspect, a hand-held device of a user signals to anautonomous vehicle that is proceeding toward a general goal location, aprecise goal location.

Implementations may include one or a combination of two or more of thefollowing features. The signaling includes line of sight signaling. Thesignaling includes sending messages through a central server. signalingincludes controlling an application running on a mobile device. Thelocation signaling activities involve a location indication signal. Theprecise goal location is the actual precise location of the user. Thesignaling includes electronic signaling or non-electronic signaling. Thenon-electronic signaling includes one or a combination of two or more ofdisplaying an image or sequence of images, emitting a sequence of lightsignals, emitting a sequence of sound signals, emitting a wirelesscommunication signal, or engaging in gestures or facial expressions. Theelectronic signaling includes signaling from one or a combination of twoor more of a smartphone, a tablet, a smart watch, smart glasses, orother smart wearable device. signaling conforms to a commonly acceptedprotocol. The precise goal location changes over time. A detector in aninfrastructure element receives signals that are part of the signalingand communicates with the autonomous vehicle or a user device based onthe signals.

These and other aspects, features, implementations, and advantages canbe expressed as methods, apparatus, systems, components, programproducts, business methods, means or steps for performing functions, andin other ways.

These and other aspects, features, implementations, and advantages willbecome apparent from the following description and from the claims.

DESCRIPTION

FIGS. 1 and 17 are block diagrams showing a process of picking up arider.

FIGS. 3, 10, 15, 16, and 18 are schematic diagrams.

FIGS. 12, 14, 19, 20, and 22 are schematic diagram related to thelocation determination process.

FIGS. 11, 13 and 21 are block diagrams related to the locationdetermination process.

FIGS. 2, 4, 5, 6, 7, 8 and 9 are display screen shots.

The use of the following terms in this description are intended broadlyand meant to include, for example, what is recited after each of theterms.

Goal location—a location that is set as a destination of the AV. The AVsystem plans a path, if one exists, and then executes that path, thattakes the AV from its current location towards the goal location. Thispath may be updated multiple times by the AV system. The goal locationmay be a general goal location or a precise goal location, and theseterms are defined below.

General location [of an object]—a location of the object (e.g., a goal,user, rider, or device) that may be specified imprecisely orinaccurately.

Precise location [of an object]—a location of the object (e.g., a goal,user, rider, device) that is specified precisely and accurately. Thestandard for what constitutes a precise location varies based on theapplication. In some applications of autonomous vehicles, a preciselocation may be one that is specified within an uncertainty of tens ofcentimetres.

Stopping place—an area that the vehicle occupies (identified by adefined shape, typically a rectangle, at a defined location in theworld) and a direction in which a vehicle is facing when stopped at thestopping place.

Target stopping place—a stopping place in the vicinity of the goallocation that is currently selected by the AV system and satisfies somepredefined notion of acceptability and feasibility.

Location indication signal—a signal that is sent by a signaling device(typically in the possession of the rider) and received by stimulusdetectors (typically on a vehicle, such as an autonomous vehicle) orother devices that is used to estimate the precise location of thesignaling device. This signal may be transmitted using a variety ofsignaling modes, such as the ones described in this application. In somecases, the location indication signal may not include an explicitindication of a location but rather may be useful in inferring orderiving a location.

As shown in FIG. 15, here we describe systems and techniques for a riderwho is to be picked up or dropped off by a vehicle, or has a parcel thatis to be picked up or dropped off by a vehicle, such as an autonomousvehicle, to indicate (signal) to the vehicle or a driver of the vehicle,a precise location that is either the precise location of the rider 1504or parcel, or a different precise location desired for the activity(pick-up or drop-off) 1508, or a different precise location nearby tothe user's actual precise location 1506. This location is termed theprecise goal location. We sometimes refer to such systems and techniquessimply as a “location signaling system.” Typically, a rider will book arequest for a pick-up or drop-off activity; the request typicallyspecifies a general goal location 1502, such as a building.

The vehicle then typically comes to a stop at a stopping place 1510 asclose to this precise goal location as possible. The user can then enterthe vehicle or deposit or pick up the parcel. In some implementations ofthe systems and techniques that we describe here, the precise goallocation is communicated (signaled) to the vehicle or the driver from adevice using one or more wireless signaling modes.

We use the phrase “rider” broadly to include, for example, any person inany context who wishes to ride in (alone or with others), hire forservice, cause a delivery to be made by, receive a delivery being madeby, or otherwise make use of a vehicle (such as an autonomous vehicle).

We use the term “hire (or hired) a vehicle for service” broadly toinclude, for example, to obtain access in any way and in any context toa vehicle for the purpose of, for example, riding in the vehicle,causing a delivery to be made by the vehicle, receiving a delivery beingmade by the vehicle, or otherwise making use of the vehicle.

Although we sometimes refer in our discussion to autonomous vehicles,the location signaling system could also be used for semi-autonomousvehicles and in some contexts for other kinds of vehicles driven inother ways. We sometimes use the term “autonomous vehicle” or simply“vehicle” to refer broadly to any such autonomous or semi-autonomous orother kind of vehicle.

Autonomous vehicles, including their components (such as sensors,actuators, and communication devices), and capabilities (such as thecapability to detect features of the environment around the vehicle andmake appropriate decisions) are detailed in U.S. patent application Ser.No. 15/240,072, filed on Aug. 18, 2016, and incorporated in its entiretyhere by reference.

We note in particular that an autonomous vehicle is capable of reactingto detected visual or other physical features in the environment,including shapes, colors, textures, movements, human gestures, flashinglights, and other visual stimuli. Autonomous vehicles may also beequipped with auditory sensors to capture and react to sounds in theenvironment. We will refer to these visual and auditory capabilities anddevices and a broad range of other detection capabilities and devicespresent in autonomous vehicles used with a location signaling system bythe general phrase “stimulus detectors.”

Conventionally, vehicles (such as taxicabs and vehicles being driven bypeople for ride-hailing services or to transport a parcel) can be hiredfor service by a rider using a ride hailing software process that mayrun on a processor unit embedded within a smartphone, laptop, tablet, orother computing device or other device. The rider typically inputs tothe ride hailing software process a ride request, which includes ageneral goal location for a pick-up activity and may also include ageneral goal location for a drop-off activity.

The general goal location can be specified by the user as anintersection (i.e. “the corner of Broadway and Main Street”), a streetaddress, or the name of a building, park, square, monument, landmark, orother general location. The user may specify the general goal locationusing software such as Google Maps, which allows users to search forlocations from various databases that store information about variouslocations. In these cases, a software process usually converts thelocation into a coordinate (latitude and longitude) using a geocodingservice, for example, Google Maps. In some cases, the rider may specifyan exact coordinate (latitude and longitude), for example, by dropping apin on a map. In some cases, the desired general goal location isautomatically identified by a software process that estimates thelocation of the device that was used to emit the position signal, usingknown geo-location methods based on GPS or other methods. The generalgoal location is therefore a representation of the location where theuser desires the pick-up or other activity take place—but it may sufferfrom the following deficiencies:

1. It may be imprecise, i.e., the representation used may notsufficiently precise for an autonomous vehicle to unambiguously knowwhere exactly in space the location lies. For example, the general goallocation may be specified as a building, which may have multiplelocations around it where an autonomous vehicle could potentially stopto perform a pick-up activity. As another example, when the general goallocation is set as the user's location at the time of the ride requestas estimated using methods such as GPS, there is often a significanterror or uncertainty associated with that estimate. For instance, GPSlocation estimates can have errors up to tens or even hundreds ofmeters.

2. It may be inaccurate, i.e., the general goal location specified inthe ride request may differ from the location where the user actuallydesires the pick-up activity to take place. This can occur due toerroneous input by the rider, uncertainty or error in the automaticidentification of the requesting device's location, uncertainty inherentin the location of the building, park, square, monument, or otherlocation, or a change of mind on the user's part after making the riderequest, or other reasons or combinations of them.

The precise goal location, on the other hand, is both precise (in thatit refers to a location in space, complying with some pre-definedstandard for precision) and accurate (in that it is assumed to representthe actual location where the rider desires that the activity takeplace). For example, the precise goal location may be specified by alatitude and longitude, whose error is restricted to be within apre-defined amount, say one square meter.

In some typical uses of autonomous vehicles, the general goal locationmay be specified by an algorithm (which may be located on a centralizedserver in the cloud and tasked with, for example, optimizing thelocations of a fleet of autonomous vehicles with a goal of minimizingrider wait times when signaling to a vehicle). For an example of aprevious scientific work related to this subject, see Spieser, Kevin, etal. “Toward a systematic approach to the design and evaluation ofautomated mobility-on-demand systems: A case study in Singapore.” RoadVehicle Automation. Springer International Publishing, 2014. 229-245.

In some cases, the general goal location may be provided by anotherprocess (e.g., an emergency process that identifies the nearest hospitalas the general goal position due to a detected medical emergency onboard the vehicle). Again, for example, the general goal location maynot indicate the actual precise location of the rider (e.g., it mayindicate the actual location of the rider imprecisely or inaccurately),due to uncertainty inherent in the algorithmically-determined generalgoal location.

In some cases, a potential rider may use a hailing device (such as asmartphone) to broadcast a hailing signal, which indicates the potentialrider's desire to hail a vehicle. This hailing signal may then bedetected and processed by stimulus detectors located on a vehicle, suchas an autonomous vehicle. Such a hailing system is described in U.S.patent application Ser. No. 15/240,072, filed on Aug. 18, 2016, andincorporated in its entirety here by reference. In such cases, thehailing signal itself could also serve as a location indication signal.When processing such a signal, in addition to processing the hailingsignal, the location indication signal may also be processed and theprecise goal position for the pick-up determined.

In known systems, as shown in FIG. 1, the general goal location of arequested ride to be provided by a vehicle having a driver iscommunicated 12 wirelessly to a centralized server tasked with assigningmovement directions to one or more, or potentially a large fleet (e.g.,hundreds or thousands) of vehicles. The centralized server then runs 14a routing software process that provides by wireless communication to aparticular vehicle (the assigned vehicle) the rider's general goallocation. The assigned vehicle then is driven 16 towards the generalgoal location to meet the rider. Often, as the assigned vehicle 1601approaches the general goal location 1602, it has difficulty meeting therider 1605, as shown in FIG. 16, because the driver has only knowledgeof the general goal location 1602, and there may be many possiblestopping places (for example, 1606, 1608 and 1610) in the vicinity ofthe general goal location. Some of these stopping places may be lesssuitable than others, because, for example, some of them are fartheraway from the rider's actual location 1604 than others. Therefore, itmay be desirable to determine a precise goal location that is suitablefor the given activity—and then find a stopping place at or in thevicinity of that precise goal location.

In order to determine a precise goal location, as shown in FIG. 15, thedriver often communicates by phone or text or both with the rider tolearn the rider's precise location 1504, or other desired precise goallocation 1508, and this communication process may be time consuming andundesirable. (This communication may be impossible in the context of anautonomous vehicle.) The driver of the vehicle then travels towards thisprecise goal location, comes to a stop at a stopping place 1510 in thevicinity of the precise goal location, and the rider enters the vehicle.Once the rider enters the vehicle and completes an identificationprocess (not necessarily in that order), the vehicle is considered hiredfor service and begins the journey to the desired drop-off location.

The flowchart of FIG. 17 and the diagram of FIG. 18 illustrate anexample pick-up process, for example, for an autonomous vehicle,employing the location signaling system described here.

Step 1 (1702): A vehicle (for example, an autonomous vehicle 1850) hastypically traveled from an initial location to a general vicinity of therider, based on a general goal location 1880 provided to the vehicle bythe rider 1802 in a ride request 1882, or by a software process 1884running on the central server, or other source.

Step 2 (1704): The rider uses a signaling device 1810 employing one or acombination of two or more of a variety of signaling modes 1820discussed below to broadcast a location indication signal 1830.

Step 3 (1706): This location indication signal is received, for example,by the stimulus detectors 1852 present on one or more nearby (moving orparked) autonomous vehicles that are within the range of the signal.

Step 4 (1708): A software process 1860 running on the autonomousvehicle, or on a central server or both as part of the AV system,analyses the information transmitted from the signaling device in thelocation indication signal, for example, by the rider and captured bythe stimulus devices, potentially in combination with road data andother information, to determine a precise goal location 1886 for therider.

Step 5 (1710): The autonomous vehicle or the AV system may send alocation response 1870 to the signaling device or the user device 1835(which may or may not be the same as the signaling device) or both, andthis response may include confirmation that the location indicationsignal was received and processed and the precise goal location that wasdetermined, among other things. This response may be received by thesignaling device or the user device or shown to the user via a userinterface 1840 or both. This response may be sent via the Internet orcellular network, or through other signaling modes that have beenestablished between the autonomous vehicle or other elements of the AVsystem and the signaling device or user device.

Step 6 (1712): The autonomous vehicle then attempts to navigate to andcome to a stop at a stopping place 1888 that is, typically, as close tothis precise goal location as possible (though other criteria may alsobe used). The rider may be kept informed of the progress of theautonomous vehicle via the signaling device or other user device.

Step 7 (1714): The rider can then enter the vehicle and complete anauthentication process (not necessarily in that order), and the vehiclecan begin its journey to the desired drop-off location.

We use the term “signaling device” broadly to include, for example, anydevice, equipment, apparatus, or system by which information about theprecise goal location for the rider's pick-up can be sent to theautonomous vehicle or to another element of the AV system. Examples ofsignaling devices are given later in the description, in conjunctionwith exemplary signaling modes. Examples of the information could be acoordinate, a pointer on a map, a position that is described relative toa known landmark or landmarks, a booking identification, a vehicleidentification, a vehicle class of service, and information about thetype of signaling device being employed (which may include informationabout the device display size and other properties.

We use the term “signaling mode” broadly to include, for example, anyform in which information can be sent by wireless signals or othermethods, including non-electronic methods. Among others, the signalingmode can include displaying an image or sequence of images, emitting asequence of light signals, emitting a sequence of sound signals,emitting a wireless communication signal, or engaging in gestures orfacial expressions, to name a few.

In most cases, the signaling device and signaling mode are designed oroperated or both to increase the incidence rate of true detections(i.e., scenarios in which the autonomous vehicle or other elements ofthe AV system correctly identifies that a signal containing informationabout a rider's precise goal location has been directed to the vehicle)and to reduce the incidence rate of false detections (i.e., scenarios inwhich an autonomous vehicle incorrectly identifies that a signalcontaining information about a rider's precise goal location has beendirected to the vehicle, when none has been sent or one has been sent toanother vehicle, for example).

The rider can signal to the autonomous vehicle using a signaling devicesuch as a mobile device, or a wide variety of other devices such assmartphones, tablets, smart wearable devices such as smart watches, orsmart glasses.

In some instances, the signaling mode may be performed by a rider (orsomeone else, such as a companion) using a portable signaling device(e.g., a smartphone). In some cases, the signaling mode may be typicalhuman capabilities performed by a rider (or someone on her behalf)without using any signaling device, but rather using other signalingmodes, such as a hand or facial expression, a head pose, or othergesture or whistling, yelling, or making another sound (or another mode)to signal a location or provide other information. In some cases, thesignaling mode may be a human signaling mode that is performedpassively, without the rider actively emitting a signal of any type, butrather by stimulus detectors on the vehicle identifying the actualprecise location of the rider using recognition of known biometricmarkers (e.g. facial recognition). Combinations of two or more of thesignaling modes can be used.

A signaling mode can comply with a predefined or commonly acceptedsignaling protocol that could specify the aspects of the locationindication signal—that is the manner in which the signaling device is tooperate or the signaling mode is to be performed, the information thatis to be transmitted, its format, and a wide variety of other aspects. Awide variety of entities that operate individual autonomous vehicles andfleets of them to be hired for service and entities that producesoftware, hardware, and other equipment for use in autonomous vehiclesand signaling devices can subscribe to such a signaling protocol. Forexample, the signaling protocol could specify the signaling mode to beused, such as a sequence of images displayed on the signaling device'sscreen. In that case, the signaling protocol could also specify theexact images that are to be displayed (or some method for generatingthose images), the size of the images, the sequence of those images, theduration for which each image in the sequence is to be displayed, thebrightness of the screen that is displaying the images, and otherrelevant features of the location indication signal. These features ofthe location indication signal can be used to encode information that isuseful in processing the signal, for instance, information related tothe booking request, or the rider, or the assigned vehicle or otherrelevant information.

A signaling device can include, for example, one or more of thefollowing (and a wide variety of others):

1. A smart phone, tablet, handheld PC, wearable device such as a smartwatch, or other configurable mobile device that is equipped with atleast a processor, a memory unit, an input device or process associatedwith a user interface, and (in some cases is equipped with) a displayscreen, light-emitting diode(s), luminescent material, e-ink screen, orother light emitting or light modulating medium. Such display andlight-emitting media may be able to exhibit or modulate sequences ofimages, colors, or lights or a combination of any two or more of themand other visual indications. The signaling device may be equipped withone or more video sensors capable of detecting visual features in theenvironment. The device may be equipped with one or more microphones orspeakers (or other sound-emitters) or both that enable the capability todetect and emit sound. The device may be equipped with wired or wirelessreceivers and transmitters that enable it to communicate with, amongother things, a central cloud or servers.

2. A printed paper or card.

3. A whistle or other noisemaker, including programmable noisemakersequipped with at least a processor and speaker, and potentially amicrophone.

Some implementations may include a signal broadcasting process 3420running on the signalling device. This process broadcasts a locationindication signal that may be an image based signal 3422, or a lightbased signal 3424, or a sound based signal 3426, or may use some othersignalling mode.

Some implementations may include a signal decoding process 3402, runningon the autonomous vehicle or a central server or some other part of theAV system, which processes a location indication signal received by astimulus detector located on an AV or other location. This process maybe able to process various aspects of the location indication signal,such as but not limited to, geometric elements 3404, colour (black andwhite, grayscale, colour) 3406, texture 3408, facial recognition 3410,gestures 3412, modulation of light 3414 and modulation of sound 3416.

Some implementations may include a location determination process 3440,running on the autonomous vehicle or a central server or some other partof the AV system, which uses the information inferred from the locationindication signal, potentially in combination with the outputs of asignal decoding process for the purpose of determining a precise goallocation. This may involve one or more of the following: bearingdetermination 3442 of the signaling device with respect to the stimulusdetector, distance determination 3442 of the signaling device withrespect to the stimulus detector, or other methods. Distancedetermination may involve a scale analysis process 3444 that analysesthe scale of an image or geometric elements in a location indicationsignal.

Signaling Modes

The signaling modes can include one or a combination of two or more ofthe following. (Generally, though not always, these signaling modes arerelevant for a rider who is signaling to a nearby autonomous vehicle,where “nearby” may be considered to be when the vehicle is, for example,within audible range of an emitted sound of moderate volume or withinline-of-sight at a range where features on the scale of a fewcentimeters might reasonably be resolved by typical sensors on anautonomous vehicle.)

Typically, it is possible to identify a relative heading from a detectedemitted signal to a relevant vehicle stimulus detector since theactivated stimulus detector elements (e.g., certain pixels in a visionsensor) are generally precisely calibrated with respect to externalstimuli.

For example, in order to identify the relative heading from a detectedemitted sound to relevant vehicle stimulus detector(s) (e.g.,microphones), a microphone array arranged in a surrounding fashionaround a vehicle central point is required. Then, the difference indetection time of the detected emitted sound by the various sensorelements in the array can be analyzed to compute the bearing from thecenter of the microphone array to the detected emitted sound usingstandard techniques.

The following are some examples of possible signaling modes.

Image Display

Signaling to a nearby autonomous vehicle may be achieved by displaying aparticular image or sequence of images (for example, displayedalternating at a fixed repeat rate) on a signaling device in a manner(e.g., presenting the device in an orientation such that the image orimages are displayed toward the roadway at shoulder height or above)that is likely to lie within a visual line of sight of, for example,video sensors mounted on a nearby autonomous vehicle.

Emitting Light

Signaling to a nearby autonomous vehicle may be achieved by emittinglight from a signaling device in a manner (e.g., presenting the devicein an orientation such that the emitted light is directed toward theroadway at shoulder height or above) that is likely to lie within avisual line of sight of video sensors mounted on an autonomous vehicle(e.g., from a sidewalk or road edge, or by a rider who is standing at anintersection).

Gesturing

Signaling to a nearby autonomous vehicle may be achieved by performing a(e.g., uncommon) gesture or sequence of gestures at a position and in anorientation that is likely to lie within visual line of sight of videoor LIDAR sensors mounted on an autonomous vehicle (e.g., from a sidewalkor road edge, or when a rider is standing at an intersection). Asoftware process running on a processor mounted on the autonomousvehicle would then analyze captured data from the video or LIDAR sensorsto detect the presence of the gesture(s) that are intended to representa position signal.

Emitting Sounds

Signaling to a nearby autonomous vehicle may be achieved by emittingfrom a signaling device (here including a smartphone or similar device,a whistle or similar device, or the rider using her mouth) an uncommonsound or sequence of sounds of sufficient volume that they can bedetected by sensors mounted on a nearby autonomous vehicle and notproduce incidences of false detection. The emitting of the sound and thefeatures and characteristics of the emitted sound can be controlled andmodulated by one of the encoding or sound modulation processes runningon a signaling device.

U.S. patent application Ser. No. 15/240,072, filed on Aug. 18, 2016, andincorporated in its entirety here by reference, provides details of thesignals, e.g., sounds, gestures, light, or images, or combinations ofthem that may be emitted, and the properties associated with thatsignal. The referenced application also details methods related toencoding information (eg. a unique ride request or booking ID) into thesignal. The referenced application refers to “haling device” and“hailing request”, which in the context of the current application,should be understood to refer to the “signaling device” and the“location indication signal” respectively.

Passive Signaling Via Biometrics-Based Identification of Riders

Location signaling to a nearby autonomous vehicle may be achievedpassively, that is, without the rider actively emitting a signal of anytype either by human behavior or through a signaling device. This can beachieved by the operation of stimulus detectors on the vehicleidentifying the rider using recognition of known biometric markers (e.g.facial recognition). In some implementations, relevant biometricinformation about a rider may have been provided explicitly to the taxi,limousine, ride-sharing, electronic hailing, or other transport service,for example, upon registration of a user account with the service.

A rider may be given an option by the transport service to provide orupdate this biometric information at the time of making a ride request(or at some other time before a pick-up or drop-off activity takesplace) to improve the performance of the recognition process. Forexample, in the case of facial recognition, a user making a ride requestfrom her smartphone may be requested to take a photo of herself usingthe camera on her smartphone, commonly known as taking a selfie, andthis can be used as the biometric marker for the facial recognitionprocess. This photo then captures the most current version of the user'sappearance, and is likely to result in better matches than performingthe recognition process on an older photo of the user, where the usermay look different. Furthermore, using a current photograph allows therecognition process to use additional features that are applicable inthe immediate situation such as the user's clothes or the background ofthe photo that could be ignored if using an older photograph as thebiometric marker.

Given relevant biometric information such as a facial image, a vastrange of known methods exist for automatic identification of specificindividuals. Typical considerations associated with such applying theknown methods that are relevant to the scenario discussed here includereliable and robust identification in variable or adverse lighting andenvironmental conditions, or at a wide variety of sensing ranges ororientations or combinations of those.

Combinations of Signaling Modes

Each of the various signaling modes that we have discussed can be usedin combination with one or more of the other modes, and with additionalmodes not mentioned explicitly. Using modes in combination can reducethe incidence rate of false detections. For example, a signaling modemay rely on executing a particular gesture (e.g., waving one's arm aboveone's head) while displaying a signaling device that is emitting aparticular temporal sequence of colors. A wide range of othercombinations of signaling modes are also possible.

Processing Received Signals to Determine a Precise Goal Location of aRider

As shown in FIG. 18, once one or more stimulus detectors 1852 on theautonomous vehicle 1850 detects a location indication signal 1830 orsignals communicated using one or more of the signaling modes 1820 andsignaling devices 1810 described above or other signaling modes orsignaling devices, a processor on the autonomous vehicle or in the AVsystem runs a position determination process 1860 that analyses theinformation contained in the location indication signal and, among otherthings, computes the precise location of the signaling device orotherwise processes the location signal to derive the precise goallocation 1886 of the rider 1802.

In some implementations, it is useful or necessary for the particularsignaling mode or modes that are used to be known a priori to both thesignal receiver and the signaling device or devices. Prior knowledge ofthe particular signaling mode or modes increases the likelihood that thestimulus detector or detectors will reliably detect emitted locationsignals, using commonly known signal detection and classificationmethods. Examples of signal detection and classification methods forvisual, auditory, gestural, and biometric-based signaling modes include,for example, supervised classification methods based on deep learning,support vector machines, neural networks, and other techniques andcombinations of them.

The process for determination of the precise goal location for a rider'spick-up can be accomplished using multiple methods, three of which aredescribed below.

FIG. 19 illustrates the term “bearing” of an object 1902 with respect toa stimulus detector 1906 on an autonomous vehicle 1904, which is used inthe methods described below. The bearing of the object 1910 with respectto the stimulus detector on the AV is defined as the angle 1912,measured in a clockwise direction, between the direction that the car isfacing (represented by the direction vector 1908) and the bearing ray1910 (which is the line joining the stimulus detector and the object).

In some implementations, the precise goal location can determined by thefollowing steps, as illustrated in FIG. 20 and the flowchart of FIG. 21:

1. The user's signaling device 600 broadcasts a location indicationsignal 610 that is detected by the stimulus detector 650 located on theautonomous vehicle 640 that is traveling in the direction 642. Thebearing 670 of the signaling device (i.e. the center of the signalingdevice, or some other point with a known relationship to the center ofthe signaling device) with respect to the stimulus detector (i.e. thecenter of the stimulus detector, or some other point with a knownrelationship to the center of the stimulus detector) may be computed asexplained previously.

2. The distance 685 from the signaling device to the stimulus detectormay be computed in multiple ways, including:

a. The distance may be computed by analysis of the detected scale of avisual feature (e.g., an image or light field) of known size that isemitted by the signaling device. For example, FIG. 22 shows a signalingdevice 600 emitting a position request using visual features 750, andthe image 760 that is captured by the stimulus detectors on theautonomous vehicle (all objects except the expected visual features havebeen filtered from the image). In some cases, a scale analysis processon the vehicle compares the size 775 of the visual feature 770 measuredfrom data produced by the stimulus detector or detectors on theautonomous vehicle to the a priori known size 755 of the visual feature750. From such a comparison, given known resolution properties of thestimulus detectors, the distance 685 from the stimulus detector ordetectors to the signaling device can be computed using known geometricanalysis methods.

This method implies that the scale analysis process running, forexample, on the vehicle has prior knowledge of the actual size of thevisual feature that is emitted by the signaling device. Such priorknowledge could be shared wirelessly at many moments during the vehiclebooking process, including at initiation of the booking request or inthe location indication signal 610, or as part of the signaling protocolthat the autonomous vehicle is aware of and complies with. Given suchprior knowledge, standard techniques can be employed to compute thescale of the visual feature as detected by stimulus detectors on theautonomous vehicle and therefore compute the distance from the stimulusdetector and the signaling device, or from other points with knownspatial relationships to the stimulus detector (for example, the centerof the autonomous vehicle) and signaling device.

b. The distance may be computed using knowledge of the bearing 670 (asdetermined, for example, in the manner described earlier) in conjunctionwith information from one or more sensors located on the vehicle. Forexample, some autonomous vehicles come equipped with LIDAR sensors whichcan send out laser light in all directions and determine with a highdegree of precision the distance to the first obstacle in the path ofeach of these beams. Given knowledge of the bearing of the signalingdevice with respect to the current location of the AV, an AV that isequipped with such a LIDAR sensor, may emit LIDAR light in the knowndirection of the signaling device. The LIDAR light would then beexpected to reflect off the signaling device and/or rider, allowing foran estimate of the distance between the AV and the signaling device.Alternatively, the LIDAR may have already emitted light in the directionof the signaling device in the recent past, and the distance may becomputed from that event. In a similar manner, given knowledge of thebearing of the signaling device, and potentially some knowncharacteristics of the signaling device or the user, other sensors suchas stereoscopic or monocular video cameras or RADAR may be used todetermine the distance of the signaling device from the AV. Such asearch may be said to be directed because of the prior knowledge of thebearing of the signaling device and is more efficient as the searchspace is reduced.

The distance may be determined repeatedly.

3. Given an estimate of the bearing 670 and distance 685 computed insteps 1 and 2, and using knowledge of the vehicle's precise location,the precise location of the signaling device 600 may be estimated. Thismay also be the precise location of the user, for example, if thesignaling device is known to be held by the user in her hands, such as,a smartphone. Alternatively, the precise position of the user may alsobe estimated if the position of the user with respect to the preciselocation of the signaling device is known (for example, if the signalingdevice is a fixed kiosk of some sort, it may be possible to detect orestimate exactly where the user is standing to use the kiosk).Typically, the AV system then sets the precise location of the signalingdevice or the rider as the precise goal location.

This method can be employed, for example, with signaling modes discussedabove that use the display of images and emission of light andpotentially other signaling modes not discussed here.

In some instances, the precise goal location can be determined by thefollowing steps, as illustrated in FIG. 14 and the flowchart of FIG. 13:

1. The user's signaling device 600 broadcasts a location indicationsignal 610 that is detected by the stimulus detector 650 located on theautonomous vehicle 640 that is traveling in the direction 642. Thebearing 670 of the signaling device (i.e., the center of the signalingdevice, or some other point with a known relationship to the center ofthe signaling device) with respect to the stimulus detector (i.e., thecenter of the stimulus detector, or some other point with a knownrelationship to the center of the stimulus detector) may be computed asexplained previously. This computation of the bearing is performedrepeatedly over a period of time (the measurement period) when thevehicle is in motion. As the vehicle moves, the bearing of the signalingdevice is likely to change. FIG. 14 shows the autonomous vehicle 640 attwo different locations at two points in time and the bearing of thesignaling device with respect to the stimulus detector is computed ateach location.

The duration of the measurement period depends on the frequency ofmeasurement (for example, if the stimulus detectors can perform ameasurement once every second, the measurement period would have to beat least 2 seconds long to acquire two measurements), the precision ofthe measurement (for example, if the measurement is not very precise,then performing a second measurement before the vehicle has moved by asubstantial distance may yield a second measurement that isstatistically indistinguishable from the first) and the number ofmeasurements needed (more measurements would require a largermeasurement period duration).

2. Using multiple measurements of the bearing, e.g., bearings 670 and671, that are computed from different positions of the vehicle as thevehicle moves during the measurement period, the precise locations ofthe signaling device 600 and the rider 620 may be estimated. Thiscomputation assumes that the rider did not move during the measurementperiod—which may be a reasonable assumption if the measurement period isrelatively small (for example, a fraction of a second)—or if the rideris moving much more slowly than the vehicle. This computation may beperformed by many known methods, for example, triangulation. In itssimplest version, triangulation uses two measurements of the bearing ofthe signaling device (for example, 670 and 671 from FIG. 14) taken fromtwo different positions of the vehicle. The signaling device is thenestimated to be located at the intersection of the two bearing rays (680and 681) which is a unique point. This method can be extended tomultiple (i.e., greater than two) measurements, for example, byestimating the precise position of the signaling device as the pointthat minimizes the sum of squared distances from that point to each ofthe bearing rays (the distance to a ray is measured perpendicularly tothe direction of that ray).

3. The precise location of the device may also be the precise locationof the user, for example, if the signaling device is known to be held bythe user in her hands, such as, a smartphone. Alternatively, theposition of the user may also be estimated if the position of the userwith respect to the signaling device is known (for example, if thesignaling device is a fixed kiosk of some sort, it may be possible todetect or estimate exactly where the user is standing to use the kiosk).Typically, the AV system then sets the precise location of the signalingdevice or the rider as the precise goal location.

This method can be employed with any of the signaling modes discussedhere, and potentially other signaling modes.

In some examples, the precise goal location can be determined in thefollowing way, as illustrated in FIG. 12 and the flowchart of FIG. 11:

1. The user's signaling device 600 broadcasts a location indicationsignal 610 that is detected by the stimulus detector 650 located on theautonomous vehicle 640 that is traveling in the direction 642. Thebearing 670 of the signaling device (i.e., the center of the signalingdevice, or some other point with a known relationship to the center ofthe signaling device) with respect to the stimulus detector (i.e., thecenter of the stimulus detector, or some other point with a knownrelationship to the center of the stimulus detector) may be computed asexplained previously.

2. Given knowledge of the bearing 670 of the signaling device, and givenroad data information related to the boundary of the drivable roadsurface 630, the intersection point 690 is computed between the bearingray 680 and the outermost (e.g., with the furthest distance from thevehicle) boundary of the drivable road surface 630 as identified in theroad data. Generally, the outermost boundary of the drivable roadsurface is a lane marking, curb, road edge, or other detected roadboundary. In some implementations, the drivable road surface may bedetermined in real-time by the AV system, potentially in conjunctionwith a central server, using a combination of static road datainformation related to the position of lane boundaries, curbs, roadedges, and information received in real-time from the AV's sensors, acentral server or other entities such as AVs, infrastructure sensors,etc. For instance, a construction zone may be present on a portion ofthe map that has previously been marked as drivable, but the AV may beable to use its perception system in real-time to detect the presence ofthis construction zone, detect the boundaries of the construction zone,and modify the drivable road surface accordingly. This updatedinformation may then be communicated back to the central server or tothe AV system or to other AVs.

3. Given knowledge of the intersection point 690 computed in step 2, itcan be set as the precise goal location. Alternatively, the precise goallocation may be identified as 697 the location of the intersection pointadjusted by a fixed offset distance 695, in order to offset theprecise-pick up location from the edge of the drivable road surface.

This last method differs from the first two methods in that it onlyestimates the bearing of the signaling device (and the user) withrespect to the autonomous vehicle and not the precise location of thesignaling device. It uses the bearing to compute a precise goal locationthat is likely, but not guaranteed, to be accurate. This method may beused, for example, when the first two methods are not implementable,e.g., if it is not possible to determine the distance between thesignaling device and the stimulus detectors, or if it is not possible tomeasure the bearing multiple times and obtain the precise user-locationusing a method such as triangulation. This method can be employed withany of the signaling modes discussed here, and potentially othersignaling modes.

The methods that we have described for determining the precise goal maybe used independently or in combination (where a final positiondetermination is computed by averaging or otherwise combining theposition estimates derived by each method individually).

It is also possible that while the rider is broadcasting a locationindication signal, the rider also moves. Therefore, an autonomousvehicle that is receiving this signal and processing it to compute aprecise location for the user (or her signaling device) and a precisegoal location for picking up that user, might receive a series oflocation indication signals, and therefore update the precise goallocation with time. Having a sequence of estimates of the preciselocation of the user is also useful in correcting for outliers and othererrors that may be inherent in the estimation process using well-knownmethods such as Kalman filtering, or hidden Markov models, and othersuch methods—and therefore improving the quality of the precise goallocation that is selected.

Having used one of the above methods for determining the precise goallocation, the AV system determines and attempts to autonomously navigateto and come to a stop at a stopping place that is feasible andacceptable and is otherwise as near to the precise goal location as ispossible, so that the rider can enter the vehicle. U.S. patentapplication Ser. No. 15/299,028, filed on Oct. 20, 2016, andincorporated in its entirety here by reference, describes the notion offeasibility and acceptability as it applies to stopping places, methodsfor an autonomous vehicle to find a feasible and acceptable stoppingplace (termed a “currently selected stopping place”) near a goallocation, and methods for an autonomous vehicle to navigate to and cometo a stop at such a target stopping place. The term “target stoppingplace” used in this application is equivalent to the term “currentlyselected stopping place” used in the above referenced application.

The stopping place that is currently selected by the AV may be updatedmultiple times before the AV comes to a stop at a target stopping place.The reasons for this are described in U.S. patent application Ser. No.15/299,028, filed on Oct. 20, 2016, and incorporated in its entiretyhere by reference. Furthermore, if the precise goal location is updatedbecause new location indication signals are received and processed bythe AV system, the AV system might also update the choice of targetstopping place. This process may continue to take place until theautonomous vehicle stops at a target stopping place.

Direct Wireless Communication

In some implementations, a precise goal location need not be determinedor inferred solely by the autonomous vehicle. The precise goal location(e.g., the precise location of the signaling device) may be determinableby the signaling device independently (i.e., without the devicebroadcasting a location indication signal to the autonomous vehicle byone or more of the signaling modes described earlier). In someinstances, the precise goal location can be determined by a cooperationof the signaling device and the autonomous vehicle.

In some instances, the signaling device may transmit its knowledge ofits precise location (or of knowledge useful in a cooperativedetermination of its precise location) to the autonomous vehicle bywireless communication of data that carries the knowledge, e.g., usingthe internet/cellular network and wireless transmitters and receiverslocated on the signaling device and the autonomous vehicle respectively.This communication may or may not happen through a central server.

Below are some examples of techniques by which the signaling device maybe able to independently determine its location:

1. The well-known GPS (Global Positioning Technology) is one example ofsuch technology. Most consumer smartphones come in-built with a GPSunit. But the position estimate obtained by most consumer-grade GPSunits may be too imprecise to be of value for the use cases described inthis application. GPS units with higher levels of precision do exist,but these are often expensive and restricted to military and scientificapplications. However, the GPS information could be useful in acooperative determination with the autonomous vehicle or the AV systemof the precise location.

2. If the signaling device included a camera (which is normally the casewith smartphones), or other visual sensor such as LIDAR—this sensor maybe used to determine the location of the device by using the sensor tocollect examples of the device's current surrounding and comparing thisto prior knowledge. For example, if the signaling device were asmartphone with a rear-facing camera, the user could be instructed toturn on the camera, and rotate a full 360 degrees without moving fromher spot, while holding the camera steady at shoulder-height. This issimilar to the method that many users follow to take panoramicphotographs from their smartphones. This method captures information onthe user's current surroundings in the form of a panoramic photograph ora video that captures vertical features of the world. This can becompared to prior information, for example in the form of a 3D model ofthe world, such as is found in Google Earth. Well known computer visiontechniques can be applied to extract features from the capturedinformation that can be compared to features extracted from the priorinformation—which allows for the estimation of the precise locationwhere the information was captured, i.e. the precise location of thesignaling device.

3. Although we have described various methods for signaling to anautonomous vehicle by sending a location indication signal from (or onbehalf of) a rider from a signaling device, similar techniques can beemployed in reverse, i.e. for the purpose of sending a signal from asignaling device located on an autonomous vehicle to stimulus detectorslocated on the user's device (e.g., smartphone). Here, the previouslydescribed visual, light-based, auditory, and gesture-based emission ofposition signals would be performed by signal emitters on the vehicleand detection would be performed by stimulus detectors present on therider's smartphone or other device. Methods for passive signaling usingbiometrics-based identification of the rider could not be employed inthis configuration, however a conceptually similar method for passivesignaling using identification of vehicle appearance could be employed(e.g., a vision sensor located on the signaling device could recognizefeatures associated with the make, model, class, type, and/orpotentially other distinguishing features, such as mechanisms or decals,placed on the autonomous vehicle solely for the purpose of enablingunique identification of the vehicle). This configuration assumes thatrelevant signal detection and emission equipment are present on therider's smartphone and on the vehicle respectively.

Once a location signal is detected by the rider's smartphone or otherdevice, the precise location of the vehicle with respect to the userdevice (or vice versa) may be calculated. Given knowledge of the preciselocation of the vehicle that is available to the AV system, the preciselocation of the user device may be inferred. This computation may takeplace on the user's smartphone, or the autonomous vehicle, or thecentral server, or on some combination of two or more of these entities.

We also note that this location signal does not need to originate fromthe rider's assigned vehicle only. Any entity that is aware of itsprecise location in space, or whose precise location is known to somecentral server (for example, based on a precise geodetic survey or otherlocation determination techniques such as LIDAR localization), and whichis equipped with the necessary signal emitters, may continuously orrepeatedly broadcast a location signal that complies with some specifiedsignaling protocol. Any device that is within range of that signal andequipped with the necessary stimulus detectors and subscribes to thatsame signaling protocol, can then interpret the location signal, and theprecise location of that device may be calculated. Examples of suchentities may include other vehicles, other autonomous vehicles, sensorsembedded into infrastructure, etc.

Communication Between an Autonomous Vehicle and a Signaling Device

We have described various methods for location signaling to anautonomous vehicle by sending a location indication signal from (or onbehalf of) a rider from a signaling device, that is, one-waycommunication from the rider to the autonomous vehicle. In some cases itmay be desirable to enable two-way communication between the rider (oran associated signaling device) and the autonomous vehicle, for thepurpose of confirming that the autonomous vehicle has receivedinformation from the signaling device, computed information related tothe rider's precise location, is traveling to the rider's precise goallocation, has come to a stop near the rider's precise goal location, orany combination of two or more of these or other reasons. The vehiclethen signals the rider as it approaches.

In some implementations, one or both of these two directions ofcommunication (between the rider and the vehicle), may be absent. Forexample, the user may specify a general goal location as part of thebooking request, the assigned vehicle may come to a stop at a stoppingplace in the vicinity of this general goal location, and the user mayfind and board the assigned vehicle without any location signalinginvolved.

As shown in FIG. 10, the user's signaling device 2702 is equipped withsignal emitters that emit location indication signals using signalingmodes 2712 that are received by stimulus detectors 2720 on theautonomous vehicle 2718. In addition, the user device and the autonomousvehicle have a two-way communication link through the internet or thecellular network. In addition to these communication interfaces, signalemitters 2724 on the autonomous vehicle may emit signals using signalingmodes 2716 that are received by stimulus detectors 2708 located on theuser device. The signaling modes 2716 may differ from the signalingmodes 2712 depending on the configuration of signal emitters andstimulus detectors on the user device and the autonomous vehicle.Similarly, these two communication links may use different signalingprotocols.

For example, to confirm that the autonomous vehicle has received thelocation indication signal from the signaling device, upon detection of,for example, an image-based, light-based, auditory, gestural, or otherlocation indication signal from a rider, the autonomous vehicle maytransmit an image-based, light-based, or auditory receipt confirmationsignal or a combination of them, for example, with the intent that thisreceipt confirmation would be uniquely identifiable as a response signalby a sensor or sensors of the rider's signaling device. The autonomousvehicle may also transmit a receipt confirmation signal through thecellular telephone network or Internet. Upon receipt of this receiptconfirmation by the rider's signaling device, the rider's signalingdevice may indicate through a user interface 2710 (e.g., using anauditory notification or visual notification on a display screen or acombination of them) to the rider that the autonomous vehicle hasidentified the precise location of the rider and is adjusting its pathin order to facilitate pick-up at the target stopping place.

Exemplary signaling modes for sending a receipt confirmation includethose already described for sending a location indication signal. Morespecifically, such methods include but are not limited to one or acombination of two or more of the following:

1. Responding to a location indication signal by display of ablack-and-white, grayscale, or color image or sequence of images on adisplay screen mounted on (for example on the outside of) the autonomousvehicle. The image properties (e.g., geometric features, texture, andappearance) should be chosen such that they can reliably be resolved bya vision sensor or sensors associated with the rider's signaling deviceand having a typical field of view and resolution, or directly perceivedby a rider, given typical human vision characteristics. The display mayalso include, or exclusively comprise, text that may be directly read bythe user or deciphered by the vision sensors on the rider's device usingoptical character recognition or other methods of reading text.

2. Responding to a location indication signal by emitting light from oneor more display screen, light emitting device, light-emitting diode, orother signaling device mounted on (for example, on the exterior of) theautonomous vehicle. The light intensity should be chosen such that itcan reliably be detected by a vision sensor or sensors or lightdetection sensor or auditory sensors associated with the rider'ssignaling device and having a typical detection sensitivity, or directlyperceived by a rider.

3. Responding to a location signal by emitting from one or more speakersmounted on (for example, on the exterior of) the vehicle a sound orsequence of sounds. The volume level should be chosen such that it canreliably be detected by a sound measurement and sensor or sensorsmounted on the rider's signaling device and having a typical detectionsensitivity, or perceived by a rider, given a typical expected ambientnoise level. Communication may also be verbal through speech output(utterances) by one or more speakers mounted on the exterior of thevehicle and verbal responses (utterances) received from the riderthrough one or more microphones mounted on the autonomous vehicle. Inorder to do so, a processor on the autonomous vehicle may execute aspeech synthesis program or dialog system contained on the vehicle, playback recorded speech, or broadcast speech received from a human remoteoperator connected to the vehicle via a wireless link. The volume may bechosen to be appropriate for the distance of the rider to the autonomousvehicle. Once a verbal response has been received by the autonomousvehicle, a speech recognizer or decoding program on the vehicle maydetermine whether to interpret the received voice signal as aconfirmation of the position signal.

4. Responding to a location signal by engaging one or more actuatorslocated on the autonomous vehicle to move objects or mechanisms locatedon the vehicle, to create movements (in effect, gestures made by thevehicle) that may be reliably resolved by a vision sensor or sensorsassociated with the rider's signaling device and having a typical fieldof view and resolution, or directly perceived by a rider, given typicalhuman vision characteristics. The rider or the target sensors associatedwith the rider's device may be provided knowledge of these mechanicalgestures. For example, if the rider used a smartphone-based hailing appto hail the AV, the app could include an animated picture or video ofthe vehicle which shows the movements being created by the actuators onthe vehicle. This way the rider is aware of the movements that thevehicle is performing and may actively look out for them.

5. Responding to a location signal by wireless communication with therider's signaling device. This could use a direct peer-to-peerconnection established between the vehicle and the rider's signalingdevice, communication through a central server which is connected toboth the vehicle and the rider's signaling device through the Internet,or communication over a cellular network.

The image or images displayed, light or lights displayed, and sound orsounds emitted by the autonomous vehicle may exhibit properties thatinclude, for example, those described above for image or imagesdisplayed, light or lights displayed, and sound or sounds emitted by thesignaling device. Information may also be encoded in the image or imagesdisplayed, light or lights displayed, and sound or sounds emitted by theautonomous vehicle using, for example, the methods described above forthe image or images displayed, light or lights displayed, and sound orsounds emitted by the signaling device.

As shown in FIG. 17, in some examples of the location signaling systemdescribed here, the autonomous vehicle first detects the locationindication signal and decodes the information embedded in it todetermine the precise goal location for the rider's pick-up. Theautonomous vehicle may then send a response to the signaling device toindicate receipt of the location indication signal. The signaling devicemay forward the response to the user by giving the user notice on amobile device using sounds or visual indicators. The autonomous vehiclethen adjusts its goal location and navigates to a stopping place nearthe rider. In some implementations, not all of the steps shown in FIG.17 need be performed.

The user is usually kept informed of the AV's current choice of astopping place and its progress in reaching that place, so that the usermay be ready to board the vehicle. As mentioned previously the targetstopping place may change with time as the AV receives new information.

The progress of the vehicle may be reported in a number of ways. Forexample, information on the precise location of the AV (which is knownto the AV) may be transmitted to the user's device and displayed on auser interface such as a map, potentially in conjunction with anestimate of the user's location (this estimate may be precise or not).Another example is to stream real-time pictures or videos taken from,for instance, a front-facing camera mounted on the AV and displayingthese to the user through a user interface on, for instance, the user'ssmartphone hailing app. These images or videos may further be annotatedto make it easier for the user to locate the vehicle using thatinformation.

FIGS. 4-9 illustrate example user interface screens that may beimplemented on a mobile device such as a tablet or a smartphone. A widevariety of other screens and features of screens can be used. The visualelements of the screens can be associated with sounds and hapticfeedback to the user of the device as part of the user interface.

FIG. 9 shows a user interface that is based on a touchscreen 2802. In atypical use case, the user would have used a hailing app (such as Uber,Lyft, Grab and many others) to book a vehicle and may track the progressof the vehicle using the hailing app. In the interface described here,the user is given the option of broadcasting her location (i.e. to sendout a location indication signal) by pressing a button 2804. This optionmay be presented in several ways, and it is also possible that thesignal is broadcast without explicitly asking the user.

The user is then presented with a choice of signaling modes 2902 asshown in FIG. 8, from which the user may pick one or more signalingmode. In some implementations, this choice may be made by the systemautomatically without explicitly asking the user. The user is thenpresented with instructions 3002, as shown in FIG. 7, that relate to thesignaling process. For example, in the case of a light or an image basedsignal that is emitted by the device display, it may be important toorient the display towards the road so that the signal travels in thedirection from which the vehicle is likely to approach. The user mayalso be instructed to remain still and hold the display at or aboveshoulder level. As another example, if the signaling was to happen usinggestures that are to be performed by the user, the instructions screenmight display these gestures to the user using pictures or videos. Theuser may then be required to confirm by pressing a button 3004 beforethe location indication signal is emitted.

As shown in FIG. 6, signaling may be achieved by displaying a particularimage or sequence of images 3104, 3106 (for example, displayedalternating at a fixed repeat rate). The image may be constructed ofpatterns 3102, or other distinctive visual features. The images may beblack and white, grayscale, or use some other defined color palette. Thetexture might include information that is encoded in a format (such asQR codes 3202 as shown in FIG. 5, which encode information in a binaryformat).

FIG. 2 illustrates an example user interface that may be implemented ona mobile device such as a tablet or a smartphone, or on a virtualreality headset (such as Google Glass or many others). A wide variety ofother screens and features of screens can be used. The visual elementsof the screens can be associated with sounds and haptic feedback to theuser of the device as part of the user interface. The figure shows anexample of an augmented reality display where the display or headsetshows the user a first person view of the world 3502 in front of theuser, which captures elements such as buildings 3504, roads 3518,vehicles 3506, and other elements in the environment. This view may beaugmented by marking the user's current location 3508, the path 3510from the user's current location to, say, the target stopping place ofthe assigned vehicle. The path may be labeled using a textual label 3512or some other graphical element. If the assigned vehicle 3516 is withinthe field of view, it may be highlighted specifically, for example usinga textual label 3514 or other graphical element. In the case of a tabletor a smartphone, the user may be instructed to hold the display at eyelevel, and a camera located behind the display may be used to record avideo of the world in front of the user. The processing capabilities ofthe device may be used to augment this video with elements such as theones described above, before displaying the augmented video on thedevice's display. In the case of a virtual reality headset, the headsetmay contain a front-facing camera and processing capabilities to achievea similar effect.

Once the location indication signal has been received and processed bythe AV system, and an appropriate precise goal location and stoppingplace have been determined, this information can be communicated back tothe user, using an interface such as the one shown in FIG. 4. Thedisplay shows a map 3306, that is focused on the area (streets may bemarked with labels such as 3302) immediately around the precise locationof the rider 3304 and target stopping place 3308.

These are marked by textual cues 3302 and 3310 so that the user clearlyunderstands that one represents her current location and the otherrepresents the stopping place where the vehicle will come and stop topick her up. A walking route 3314 from the precise location of the userto the stopping place is also shown. The map may also show the precisevehicle location 3312, though the vehicle may not be visible to the userif it is not close to the user. The interface may also show details 3322of the assigned vehicle, such as the vehicle registration number, modeland make, color and other relevant details. The map interface maysupport standard navigational techniques such as panning and zooming inand out. Finally, the user is also provided an option of changing thestopping place by pressing a button 3316. This would allow the user tochoose from a list of stopping places (that has been curated by the AVsystem) as described in U.S. patent application Ser. No. 15/299,028,filed on Oct. 20, 2016, and incorporated in its entirety here byreference (see FIG. 13 in the referenced patent application).

Because the target stopping place may change multiple times before thevehicle comes to a stop at a target stopping place, it may be desirableto avoid showing the user the target stopping place, until the vehiclehas actually stopped in it, or is fairly confident that it will be ableto stop in it (for example, when the vehicle is very close to the targetstopping place and has verified with its own sensors that it isavailable.)

Communication without Line-of-Sight Using Sensors Located inInfrastructure or in Other Vehicles

The signaling modes described thus far involve direct communicationbetween the signaling device and the stimulus detectors located on theassigned vehicle or the rider's device or both. Therefore, for thesemodes to function, the assigned vehicle's stimulus detectors must bewithin a certain range of the signaling device, and for visual stimuli,no physical obstructions may be present between the signaling device andthe stimulus detector.

It is possible that the location indication signals being broadcast by asignaling device are received by a vehicle other than the assignedvehicle because that vehicle is within range of the signaling device.This may happen because the vehicle is scanning for and able to receivesignals other than from its assigned passenger's signaling device. Thecentral server may also deliberately instruct all vehicles in its fleetthat are within a certain distance of the rider's general location toscan for and detect location indication signals from the rider'ssignaling device. The purpose of such a strategy is to increase thelikelihood of detecting the rider's location indication signal ordetecting the location indication signal sooner. The vehicle thatdetected the location signal may then compute the precise location ofthe rider, or a precise goal location for the pick-up, using the methodsdescribed previously, and transmit this information to the rider'sassigned vehicle using the cellular network or the Internet.

To increase the effective range of the signaling device, stimulusdetectors equipped with computing devices and with access to map datamay be embedded in infrastructure. Examples of such infrastructureinclude streetlamps, public telephones or kiosks, CCTV cameras,pavements, curbs etc. These may include stimulus detectors similar tothose described previously and they may receive location indicationsignals from the rider's signaling device. The computation of theprecise location of the rider or the precise goal location may takeplace on these devices, or the information from the signal may berelayed to a central server or an autonomous vehicle or the user deviceor other location where the computation takes place using the cellularnetwork or the Internet, or the computation may take place cooperativelybetween one or more of such entities. This information may then berelayed to the assigned vehicle via the cellular network or theInternet.

Teleoperator-Assisted Location Determination

Determination of the precise location of a user or a precise goallocation may also be performed by a remote operator, also known as atele-operator, who may be presented with the signals received by thestimulus detectors, in a raw or a processed form, in addition to otherdata. This may happen, for example, when the location determinationprocess is unable to estimate the precise location of the user withsufficient precision, or a tele-operator may be used to process alllocation indication signals.

For example, in the case of signaling modes such as an image baseddisplay, or gestures, or passive biometric recognition of the rider'sface, where the stimulus detector is a camera (or multiple cameras)located on the autonomous vehicle, the video stream from these camerasmay be transmitted to the tele-operator. These video streams may bepresented to the tele-operator on a user interface, such as atouchscreen monitor or other display device. The video streams may bepresented to the tele-operator as-is, or they may be augmented, forexample using the processes and algorithms discussed previously in thisapplication, to aid the tele-operator by attempting to determine theprecise location of the user and providing that to the tele-operator asa suggestion. The video streams may also be augmented by overlaying datafrom other sensors over the video, for example a point-cloud from aLIDAR sensor. The user interface may also permit the tele-operator toidentify the rider, or the precise location of the rider, or a precisegoal location for the autonomous vehicle by providing a touch-basedinterface to the tele-operator, for example, by clicking on atouchscreen.

Other implementations are within the scope of the following claims.

The invention claimed is:
 1. A method comprising: receiving, by anautonomous vehicle system, a request for use of an autonomous vehiclefrom a user, the request including in indication of a general goallocation, the general goal location having a first degree of precisionand a first degree of accuracy regarding a location of the user;instructing, by the autonomous vehicle system, the autonomous vehicle toproceed from an initial location towards the general goal location;while the autonomous vehicle is proceeding from the initial locationtowards the general goal location, determining, by the autonomousvehicle system, a precise goal location, the precise goal location beingwithin a vicinity of a general goal location, the precise locationhaving a second degree of precision and a second degree of accuracyregarding the location of the user, wherein at least one of: the seconddegree of precision is greater than the first degree of precision, orthe second degree of accuracy is greater than the first degree ofaccuracy, and wherein the precise goal location is determined based on awireless communication between the autonomous vehicle and the user whenthe autonomous vehicle and the user are within line-of-sight;determining, by the autonomous vehicle system, a stopping place at whichthe autonomous vehicle and a user will engage in a destination goallocation activity, the stopping place being within a vicinity of theprecise goal location; and instructing, by the autonomous vehiclesystem, the autonomous vehicle to proceed towards and stop at thestopping place.
 2. The method of claim 1, wherein the precise goallocation is the actual precise location of the user.
 3. The method ofclaim 1, wherein the precise goal location is determined based on adistance from the autonomous vehicle to one of the precise goal locationand the general goal location.
 4. The method of claim 3, wherein thedistance is determined comprising determining the distance repeatedly.5. The method of claim 1, wherein the precise goal location isdetermined based on road data.
 6. The method of claim 1, wherein theprecise goal location changes over time.
 7. The method of claim 1,further comprising: transmitting, by the autonomous vehicle system to amobile device associated with the user, a command to generate an imageor series of images using a display of the mobile device, and whereindetermining the precise goal location comprises: obtaining, by theautonomous vehicle system, sensor data comprising visual informationregarding an environment of the autonomous vehicle, identifying, by theautonomous vehicle system based on the sensor data, the image or theseries of images in the environment of the autonomous vehicle, anddetermining the precise goal location based on the identification of theimage or the series of images in the environment of the autonomousvehicle.
 8. The method of claim 1, further comprising: transmitting, bythe autonomous vehicle system to a mobile device associated with theuser, a command to generate a sound using a speaker of the mobiledevice, and wherein determining the precise goal location comprises:obtaining, by the autonomous vehicle system, sensor data comprisingauditory information regarding an environment of the autonomous vehicle,identifying, by the autonomous vehicle system based on the sensor data,the sound in the environment of the autonomous vehicle, and determiningthe precise goal location based on the identification of the sound inthe environment of the autonomous vehicle.
 9. The method of claim 1,wherein determining the precise goal location comprises: obtaining, bythe autonomous vehicle system, sensor data comprising visual informationregarding an environment of the autonomous vehicle, identifying, by theautonomous vehicle system based on the sensor data, the user performinga pre-determined gesture in in the environment of the autonomousvehicle, and determining the precise goal location based on theidentification of the user performing the pre-determined gesture in theenvironment of the autonomous vehicle.
 10. The method of claim 1,further comprising: transmitting, by the autonomous vehicle system to amobile device associated with the user, a command to generate amodulated sequence of light using the mobile device, and whereindetermining the precise goal location comprises: obtaining, by theautonomous vehicle system, sensor data comprising visual informationregarding an environment of the autonomous vehicle, identifying, by theautonomous vehicle system based on the sensor data, the modulatedsequence of light in the environment of the autonomous vehicle, anddetermining the precise goal location based on the identification of themodulated sequence of light in the environment of the autonomousvehicle.
 11. The method of claim 1, wherein determining the precise goallocation comprises: determining, based on the wireless communication, abearing of a user relative to the autonomous vehicle; responsive todetermining the bearing of a user relative to the autonomous vehicle,instructing the autonomous vehicle to direct a directional sensortowards the determined bearing; and determining a distance between theuser and the autonomous vehicle based on sensor data obtained from thedirectional sensor.